
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/535,441 



03/23/2000 



21876 7590 01/05/2004 

FISH & RICHARDSON P.C. 
3300 DAIN RAUSCHER PLAZA 
MINNEAPOLIS, MN 55402 



Chinmoy Panda 



07844-364001 



1295 



EXAMINER 



SMITH, PETER J 



ART UNIT 



PAPER NUMBER 



2176 

DATE MAILED: 01/05/2004 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 



Application No. 

09/535,441 



Examiner 

Peter J Smith 



Applicant(s) 

PANDA, CHINMOY 



Art Unit 

2176 



The MAILING DATE of this communication appears on the cover sheet with the correspondence address -- 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 
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- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )^ Responsive to communication(s) filed on 27 October 2003 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-13 and 27-41 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) KI Claim(s) 1-13 and 27-41 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10)^ The drawing(s) filed on 23 March 2000 is/are: a)M accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .121 (d). 
1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
Priority under 35 U.S.C. §§ 119 and 120 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a>n All bO Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

13) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1.78. 

a) D The translation of the foreign language provisional application has been received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1 .78. 
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DETAILED ACTION 



1 . This action is responsive to communications: amendment filed on 10/27/2003, 
application filed on 03/23/2000. 

2. The rejection of claims 1-2, 6, 8-13, and 27-28 under 35 U.S.C. 103(a) as being 
unpatentable over Ho et al., US 6,021,412 filed 04/02/1996 in view of Sciammarella et al., US 
5,982,369 filed 04/21/1997 has been dropped as necessitated by amendment. 

3. The rejection of claims 3-5 under 35 U.S.C. 103(a) as being unpatentable over Ho et al., 
US 6,021,412 filed 04/02/1996 in view of Sciammarella et al., US 5,982,369 filed 04/21/1997 as 
applied to claims 1 and 14 above, and further in view of Dutta, US 6,480,837 Bl filed 
12/16/1 999 has been dropped as necessitated by amendment. 

4. The rejection of claim 7 under 35 U.S.C. 103(a) as being unpatentable over Ho et al., US 
6,021,412 filed 04/02/1996 in view of Sciammarella et al, US 5,982,369 filed 04/21/1997 as 
applied to claims 6 and 19 above, and further in view of Matsumoto, US 6,526,170 filed 
12/13/1994 has been dropped as necessitated by amendment. 

5. Claims 14-26 have been cancelled. 

6. Claims 1-13 and 27-41 are pending in the case. Claim 1, 27, and 29 are independents 
claim. 

Claim Rejections - 35 USC§103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 



(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
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having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

7. Claims 1-2, 6, 8-13, 27-28, 29-30, 34, and 36-41 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over Ho et al. (hereafter referred to as Ho), US 6,021,412 filed 
04/02/1996 in view of Sciammarella et al. (hereafter referred to as Sciammarella), US 
5,982,369 filed 04/21/1997 and Rosenbaum, US 5,404,435 patented 04/04/1995. 

Regarding independent claims 1 and 29, Ho teaches extracting one or more document 
keywords from a document in col. 1 lines 50-55. Ho also teaches collecting one or more images 
associated with the document including information describing each image in col. 1 line 55 - col. 
2 line 5. Ho does not teach is generating a proximity factor for each image collected from the 
document and each document keyword that reflects the degree of correlation between the image 
and the document keyword or determining the importance of each image according to an image 
metric that combines the proximity factors for each document keyword and image pair. Ho does 
not teach that each image has a location in the document. 

Sciammarella does teach generating a proximity factor for each pair of the one or more 
images and the one or more document keywords, the proximity factor reflecting a degree of 
correlation between the image and the document keyword of the pair in Fig. 2 and the abstract. 
Sciammarella also teaches determining the importance of each image according to an image 
metric that combines the proximity factors for each document keyword and image pair in Fig. 4 
and the abstract. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the invention as claimed. It would have 
been obvious and desirable to one of ordinary skill in the art at the time of the invention to 
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generate a proximity factor relationship between the images and keywords so that user could 
discern between the different relational importance of each keyword and image pair. This would 
have allowed the user to have an additional means for choosing the best image for a particular 
location on the document. 

Rosenbaum teaches collecting non-text objects, such as image objects, from a document 
in the abstract and col. 1 line 67 - col. 2 line 2. It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to have combined Rosenbaum into Ho to have 
created the claimed invention. It would have been obvious to have collected the images from 
current documents in possession of the user so that they could be used to assist the user in 
creating future documents. 

Regarding dependent claims 2 and 30, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. 

Regarding dependent claims 6 and 34, Ho teaches identifying image text associated 
with each image of the one or more images in Fig. 5 item 506 and Fig. 6. 

Regarding dependent claims 8 and 36, Ho teaches searching the metadata information 
associated with the image for text describing the image in Fig. 5 item 506 and col. 1 lines 47-50. 

Regarding depend claims 9 and 37, Ho does not explicitly teach that the metadata 
information is compatible with hypertext markup language. One of ordinary skill in the art at the 
time of the invention would have known how to make metadata compatible with hypertext 
markup language. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine the their knowledge with Ho to create the claimed invention. It 
would have been obvious and desirable to use hypertext markup language compatible metadata 
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because hypertext markup language is the most common form of storing metadata and it would 
have been advantageous to use it to create an invention compatible with well known standards. 

Regarding dependent claims 10 and 38, Ho does not explicitly teach lexically 
analyzing the image text associated with the image and the keyword to determine the degree of 
correlation between the image and the keyword. Sciammarella does teach lexically analyzing the 
image text associated with the image and the keyword to determine the degree of correlation 
between the image and the keyword in col. 2 lines 26-30. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the claimed invention. It would have been 
obvious and desirable to determine the degree of correlation between the keywords and the 
images to best determine what image should be placed in each location. 

Regarding dependent claims 11 and 39, Ho does not teach performing a phonetic 
comparison between the image text associated with the image and the document keyword to 
determine the degree of correlation between the image and the document keyword. 
Sciammarella does teach performing a phonetic comparison between the image text associated 
with the image and the document keyword to determine the degree of correlation between the 
image and the document keyword in Fig. 2 and col. 2 lines 2-18. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the claimed invention. It would have been 
obvious and desirable to determine the degree of correlation between each image and keyword 
pair to determine which image is the best match for a particular keyword. 
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Regarding dependent claims 12 and 40, Ho teaches identifying the location of the 
image in the document in col. 5 lines 14-17 and measuring the distance between the image in the 
document and a keyword in col. 6 lines 57-59 as the program knows the positions of both the 
graphic and the keyword on the page. 

Regarding dependent claims 13 and 41, Ho teaches determining the correlation 
between each document keyword and an image according to the distance between the document 
keyword and the image in col. 5 lines 14-17. 

Regarding independent claim 27, Ho teaches extracting one or more document 
keywords from a document in col. 1 lines 50-55. Ho also teaches collecting one or more images 
associated with the document including information describing each image in col. 1 line 55 - col. 
2 line 5. Ho does not teach is generating a proximity factor for each image collected from the 
document and each document keyword that reflects the degree of correlation between the image 
and the document keyword or determining the importance of each image according to an image 
metric that combines the proximity factors for each document keyword and image pair. Ho does 
not teach that each image has a location in the document. 

Sciammarella does teach generating a proximity factor for each pair of the one or more 
images and the one or more document keywords, the proximity factor reflecting a degree of 
correlation between the image and the document keyword of the pair in Fig. 2 and the abstract. 
Sciammarella also teaches determining the importance of each image according to an image 
metric that combines the proximity factors for each document keyword and image pair in Fig. 4 
and the abstract. 
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It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Sciammarella into Ho to create the invention as claimed. It would have 
been obvious and desirable to one of ordinary skill in the art at the time of the invention to 
generate a proximity factor relationship between the images and keywords so that user could 
discern between the different relational importance of each keyword and image pair. This would 
have allowed the user to have an additional means for choosing the best image for a particular 
location on the document. 

Rosenbaum teaches collecting non-text objects, such as image objects, from a document 
in the abstract and col. 1 line 67 - col. 2 line 2. It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to have combined Rosenbaum into Ho to have 
created the claimed invention. It would have been obvious to have collected the images from 
current documents in possession of the user so that they could be used to assist the user in 
creating future documents. 

Regarding dependent claim 28, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. 

8. Claims 3-5 and 31-33 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ho et al. (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in view of 
Sciammarella et al. (hereafter referred to as Sciammarella), US 5,982,369 filed 04/21/1997 
and Rosenbaum, US 5,404,435 patented 04/04/1995 as applied to claims 1 and 29 above, 
and further in view of Dutta, US 6,480,837 Bl filed 12/16/1999. 
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Regarding dependent claims 3 and 31, Ho does not teach ordering the document 
keywords according to an ordering criterion or weighting the proximity factor associated with 
each document keyword and image pair based on the order of the document keyword. Dutta 
does teach ordering the document keywords according to an ordering criterion and weighting the 
proximity factor associated with each document keyword and image pair based on the order of 
the document keyword in Fig. 2. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the claimed invention. It 
would have been obvious and desirable to order and weight the keywords according to a 
popularity weight so that some of the keywords would have had an increased chance of gaining 
an image located near them. This would have been desirable to give the user an ability to 
accentuate certain parts of the document over others with the use of images. 

Regarding dependent claims 4 and 32, Ho does not teach that the frequency that each 
document keyword appears in the document determines the ordering criterion used to order the 
document keywords. Dutta does teach that the frequency that each document keyword appears 
in the document determines the ordering criterion used to order the document keywords in col. 1 
lines 23-37. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the invention as claimed. 
It would have been obvious and desirable to use the frequency of the keywords to determine the 
location the image should be placed on the document. It is common sense to place the image 
where the keyword appears most frequently. 
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Regarding dependent claims 5 and 33, Ho does not teach that the ordering criterion 
orders the document keywords according to their relationship with the subject matter of the 
document. Dutta does teach that the ordering criterion orders the document keywords according 
to their relationship with the subject matter of the document in col. 4 lines 25-42. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Sciammarella to create the claimed invention. It 
would have been obvious and desirable to order the keywords according to the relationship with 
the subject matter so that the images are matched with the keywords with which they have the 
strongest relationship. 

9. Claims 7 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ho 
et al. (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in view of Sciammarella et 
al. (hereafter referred to as Sciammarella), US 5,982,369 filed 04/21/1997 and Rosenbaum, 
US 5,404,435 patented 04/04/1995 as applied to claims 6 and 34 above, and further in view 
of Matsumoto, US 6,526,170 filed 12/13/1994. 

Regarding dependent claims 7 and 35, Ho does not teach scanning a bit-mapped 
representation of the image for text information or converting the bit-mapped representation of 
the text information into image text. Matsumoto does teach scanning a bit-mapped 
representation of the image for text information and converting the bit-mapped representation of 
the text information into image text in Fig. 7 and col. 1 lines 23-25. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Matsumoto into Ho in view of Sciammarella to create the claimed 
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invention. It would have been obvious and desirable to extract text from the image itself and use 
that to match with the keywords so that the image-keyword match would further increase in 
accuracy. 

Response to Arguments 
10. Applicant's arguments filed 10/27/2003 have been fully considered but they are not 
persuasive. Regarding Applicant's argument in Section I, paragraphs 1-4 that neither Ho nor 
Sciammarella teach collecting image having a location in a document or generating a proximity 
factor for a pair of an image and a document keyword, Ho teaches matching an image to concept 
keywords in fig. 5 and 6, and col. 1 line 38 - col. 2 line 5. Sciammarella additionally teaches 
matching a selection of images with keywords with each pair having a proximity factor in fig. 4 
and the abstract. The Examiner has additionally incorporated the reference of Rosenbaum, 
which teaches retrieval of non-text objects, such as image objects (see abstract), from mixed 
object documents and stores the image objects for future retrieval and use. The Examiner 
believe the combination of these three references renders the claimed invention obvious. 

Regarding Applicant's argument in Section I, paragraph 5 that neither Ho nor 
Sciammarella use images having location in a document. Again, the Examiner has additionally 
incorporated the reference of Rosenbaum, which teaches retrieval of non-text objects, such as 
image objects (see abstract), from mixed object documents and stores the image objects for 
future retrieval and use. The Examiner believe the combination of these three references renders 
the claimed invention obvious. 
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Regarding Applicant's argument in Section I, paragraph 6 that neither Ho nor 
Sciammarella teach lexically analyzing image text and phonetic comparison between the image 
text and the keyword, measuring the distance between the image and the keyword, or generating 
a proximity factor for an image and document keyword and image pair, Sciammarella teaches 
lexically analyzing image text and phonetic comparison between the image text and the keyword 
in fig. 2, col. 2 lines 26-30, and col. 2 lines 2-18. Sciammarella teaches measuring the distance 
between the image and the keyword in fig. 2, fig. 4 and col. 2 lines 26-30. Sciammarella teaches 
generating a proximity factor for an image and document keyword and image pair in fig. 2, col. 2 
lines 26-30, and col. 2 lines 2-18. 



11. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 



Conclusion 
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12. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 703-305-5931 . The 
examiner can normally be reached on Mondays-Fridays 7:00am-3 :30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 703-305-9792. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-305-3900. 



PJS 

December 24, 2003 




